<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns# article: http://ogp.me/ns/article# " lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>python-parallel-system-tools-pp4e | 绿萝间</title>
<link href="../assets/css/all-nocdn.css" rel="stylesheet" type="text/css">
<link href="../assets/css/ipython.min.css" rel="stylesheet" type="text/css">
<link href="../assets/css/nikola_ipython.css" rel="stylesheet" type="text/css">
<meta name="theme-color" content="#5670d4">
<meta name="generator" content="Nikola (getnikola.com)">
<link rel="alternate" type="application/rss+xml" title="RSS" href="../rss.xml">
<link rel="canonical" href="https://muxuezi.github.io/posts/python-parallel-system-tools-pp4e.html">
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
    tex2jax: {
        inlineMath: [ ['$','$'], ["\\(","\\)"] ],
        displayMath: [ ['$$','$$'], ["\\[","\\]"] ],
        processEscapes: true
    },
    displayAlign: 'center', // Change this to 'center' to center equations.
    "HTML-CSS": {
        styles: {'.MathJax_Display': {"margin": 0}}
    }
});
</script><!--[if lt IE 9]><script src="../assets/js/html5.js"></script><![endif]--><meta name="author" content="Tao Junjie">
<link rel="prev" href="kivy-ch2-paint-app.html" title="kivy-ch2-paint-app" type="text/html">
<link rel="next" href="kivy-ch3-sound-recorder-for-android.html" title="kivy-ch3-sound-recorder-for-android" type="text/html">
<meta property="og:site_name" content="绿萝间">
<meta property="og:title" content="python-parallel-system-tools-pp4e">
<meta property="og:url" content="https://muxuezi.github.io/posts/python-parallel-system-tools-pp4e.html">
<meta property="og:description" content="Parallel System Tools in Programming Python, 4th Edition, include forks,
threads, pipes, signals, sockets, and other launching techniques.









Forking Processes¶








1.fork¶





In [1]:

   ">
<meta property="og:type" content="article">
<meta property="article:published_time" content="2015-06-24T13:42:41+08:00">
<meta property="article:tag" content="Parallel Programming with Python">
<meta property="article:tag" content="Python">
</head>
<body>
<a href="#content" class="sr-only sr-only-focusable">Skip to main content</a>

<!-- Menubar -->

<nav class="navbar navbar-inverse navbar-static-top"><div class="container">
<!-- This keeps the margins nice -->
        <div class="navbar-header">
            <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-navbar" aria-controls="bs-navbar" aria-expanded="false">
            <span class="sr-only">Toggle navigation</span>
            <span class="icon-bar"></span>
            <span class="icon-bar"></span>
            <span class="icon-bar"></span>
            </button>
            <a class="navbar-brand" href="https://muxuezi.github.io/">

                <span id="blog-title">绿萝间</span>
            </a>
        </div>
<!-- /.navbar-header -->
        <div class="collapse navbar-collapse" id="bs-navbar" aria-expanded="false">
            <ul class="nav navbar-nav">
<li>
<a href="../archive.html">Archive</a>
                </li>
<li>
<a href="../categories/">Tags</a>
                </li>
<li>
<a href="../rss.xml">RSS feed</a>

                
            </li>
</ul>
<ul class="nav navbar-nav navbar-right">
<li>
    <a href="python-parallel-system-tools-pp4e.ipynb" id="sourcelink">Source</a>
    </li>

                
            </ul>
</div>
<!-- /.navbar-collapse -->
    </div>
<!-- /.container -->
</nav><!-- End of Menubar --><div class="container" id="content" role="main">
    <div class="body-content">
        <!--Body content-->
        <div class="row">
            
            
<article class="post-text h-entry hentry postpage" itemscope="itemscope" itemtype="http://schema.org/Article"><header><h1 class="p-name entry-title" itemprop="headline name"><a href="#" class="u-url">python-parallel-system-tools-pp4e</a></h1>

        <div class="metadata">
            <p class="byline author vcard"><span class="byline-name fn">
                    Tao Junjie
            </span></p>
            <p class="dateline"><a href="#" rel="bookmark"><time class="published dt-published" datetime="2015-06-24T13:42:41+08:00" itemprop="datePublished" title="2015-06-24 13:42">2015-06-24 13:42</time></a></p>
            
        <p class="sourceline"><a href="python-parallel-system-tools-pp4e.ipynb" id="sourcelink">Source</a></p>

        </div>
        

    </header><div class="e-content entry-content" itemprop="articleBody text">
    <div tabindex="-1" id="notebook" class="border-box-sizing">
    <div class="container" id="notebook-container">

<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Parallel System Tools in <a href="http://www.rmi.net/~lutz/about-pp4e.html">Programming Python, 4th Edition</a>, include forks,
threads, pipes, signals, sockets, and other launching techniques.</p>
<!-- TEASER_END -->
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Forking-Processes">Forking Processes<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#Forking-Processes">¶</a>
</h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="1.fork">1.fork<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.fork">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">child</span><span class="p">():</span>
    <span class="nb">print</span> <span class="s2">"hello from child"</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">()</span>
    <span class="n">os</span><span class="o">.</span><span class="n">_exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># else goes back to parent loop</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">parent</span><span class="p">():</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="c1">#generates a copy of the calling program, </span>
        <span class="c1"># it returns a different value in each copy: </span>
        <span class="c1"># zero in the child process and </span>
        <span class="c1"># the process ID of the new child in the parent.</span>
        <span class="n">newpid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span> 
        <span class="k">if</span> <span class="n">newpid</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
            <span class="n">child</span><span class="p">()</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="nb">print</span> <span class="s2">"hello from parent"</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">newpid</span>
        <span class="k">if</span> <span class="n">raw_input</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'q'</span><span class="p">:</span> <span class="k">break</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [8]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">parent</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>hello from parent 8118 8360

hello from parent 8118 8366
hello from child 8360

hello from parent 8118 8373
hello from child 8366
q
hello from child 8373
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="2.fork-count">2.fork-count<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.fork-count">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [15]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span><span class="nn">time</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [27]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">counter</span><span class="p">(</span><span class="n">count</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
        <span class="nb">print</span> <span class="s2">"[</span><span class="si">%s</span><span class="s2">] =&gt; </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">i</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [30]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
    <span class="n">pid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span>
    <span class="k">if</span> <span class="n">pid</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
        <span class="nb">print</span> <span class="s2">"Process </span><span class="si">%d</span><span class="s2"> spawned"</span> <span class="o">%</span> <span class="n">pid</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">counter</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
        <span class="n">os</span><span class="o">.</span><span class="n">_exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="nb">print</span> <span class="s2">"Main process exiting."</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Process 9008 spawned
Process 9009 spawned
Process 9010 spawned
Process 9011 spawned
Process 9012 spawned
Main process exiting.
[8979] =&gt; 0
[8978] =&gt; 0
[8980] =&gt; 0
[8981] =&gt; 0
[8982] =&gt; 0
[8979] =&gt; 1[8978] =&gt; 1[8980] =&gt; 1[8981] =&gt; 1[8982] =&gt; 1




</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="3.The-fork/exec-Combination">3.The fork/exec Combination<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#3.The-fork/exec-Combination">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [32]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [34]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">parm</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
    <span class="n">parm</span> <span class="o">+=</span> <span class="mi">1</span>
    <span class="n">pid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span>
    <span class="k">if</span> <span class="n">pid</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
        <span class="n">os</span><span class="o">.</span><span class="n">execlp</span><span class="p">(</span><span class="s1">'python'</span><span class="p">,</span><span class="s1">'python'</span><span class="p">,</span><span class="s1">'child.py'</span><span class="p">,</span><span class="nb">str</span><span class="p">(</span><span class="n">parm</span><span class="p">))</span>
        <span class="k">assert</span> <span class="kc">False</span><span class="p">,</span> <span class="s1">'error starting program'</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="nb">print</span> <span class="s2">"Child is"</span><span class="p">,</span> <span class="n">pid</span>
        <span class="k">if</span> <span class="n">raw_input</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'q'</span><span class="p">:</span> <span class="k">break</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Child is 9654

Child is 9655

Child is 9656

Child is 9657
q
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Threads">Threads<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#Threads">¶</a>
</h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="1.The-thread-Module">1.The thread Module<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.The-thread-Module">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.1Basic-usage">1.1Basic usage<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.1Basic-usage">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [36]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">thread</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [37]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">child</span><span class="p">(</span><span class="n">tid</span><span class="p">):</span>
    <span class="nb">print</span> <span class="s1">'Hello from thread'</span><span class="p">,</span> <span class="n">tid</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [38]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">parent</span><span class="p">():</span>
    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,))</span>
        <span class="k">if</span> <span class="n">raw_input</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'q'</span><span class="p">:</span> <span class="k">break</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [39]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">parent</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>
Hello from thread 1

Hello from thread 2
q
Hello from thread 3
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.2-Other-ways-to-code-threads-with-thread">1.2 Other ways to code threads with thread<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.2-Other-ways-to-code-threads-with-thread">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [40]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">thread</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [42]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">action</span><span class="p">(</span><span class="n">i</span><span class="p">):</span>
    <span class="nb">print</span> <span class="n">i</span><span class="o">**</span><span class="mi">32</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [43]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">class</span> <span class="nc">Power</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">i</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">i</span> <span class="o">=</span> <span class="n">i</span>
    <span class="k">def</span> <span class="nf">action</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="nb">print</span> <span class="bp">self</span><span class="o">.</span><span class="n">i</span><span class="o">**</span><span class="mi">32</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [51]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">action</span><span class="p">,(</span><span class="mi">2</span><span class="p">,))</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt output_prompt">Out[51]:</div>


<div class="output_text output_subarea output_execute_result">
<pre>140621501368064</pre>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [52]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="k">lambda</span><span class="p">:</span> <span class="n">action</span><span class="p">(</span><span class="mi">2</span><span class="p">),</span> <span class="p">())</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>4294967296
</pre>
</div>
</div>

<div class="output_area">
<div class="prompt output_prompt">Out[52]:</div>


<div class="output_text output_subarea output_execute_result">
<pre>140621501368064</pre>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [56]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">obj</span> <span class="o">=</span> <span class="n">Power</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [57]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">obj</span><span class="o">.</span><span class="n">action</span><span class="p">,())</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt output_prompt">Out[57]:</div>


<div class="output_text output_subarea output_execute_result">
<pre>140621501368064</pre>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.3-Running-multiple-threads">1.3 Running multiple threads<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.3-Running-multiple-threads">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [59]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">thread</span><span class="o">,</span><span class="nn">time</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [63]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">counter</span><span class="p">(</span><span class="n">myId</span><span class="p">,</span> <span class="n">count</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
        <span class="nb">print</span> <span class="s1">'[</span><span class="si">%s</span><span class="s1">] =&gt; </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">myId</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [48]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
    <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">counter</span><span class="p">,(</span><span class="n">i</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [49]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="nb">print</span> <span class="s1">'Main threading exit.'</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>[2] =&gt; 0
[4] =&gt; 0
[1] =&gt; 0
[0] =&gt; 0
[3] =&gt; 0
[2] =&gt; 1
[4] =&gt; 1
[1] =&gt; 1
[0] =&gt; 1
[3] =&gt; 1
[2] =&gt; 2
[4] =&gt; 2
[1] =&gt; 2
[0] =&gt; 2
[3] =&gt; 2
Main threading exit.
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.4-Synchronizing-access-to-shared-objects-and-names">1.4 Synchronizing access to shared objects and names<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.4-Synchronizing-access-to-shared-objects-and-names">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [32]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">thread</span><span class="o">,</span><span class="nn">time</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [33]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">counter</span><span class="p">(</span><span class="n">myId</span><span class="p">,</span> <span class="n">count</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
        <span class="n">mutex</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
        <span class="nb">print</span> <span class="s1">'[</span><span class="si">%s</span><span class="s1">] =&gt; </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">myId</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
        <span class="n">mutex</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [45]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">mutex</span> <span class="o">=</span> <span class="n">thread</span><span class="o">.</span><span class="n">allocate_lock</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
    <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">counter</span><span class="p">,(</span><span class="n">i</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [46]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="nb">print</span> <span class="s1">'Main threading exit.'</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>[0] =&gt; 0
[1] =&gt; 0
[2] =&gt; 0
[3] =&gt; 0
[4] =&gt; 0
[0] =&gt; 1
[1] =&gt; 1
[2] =&gt; 1
[3] =&gt; 1
[4] =&gt; 1
[0] =&gt; 2
[1] =&gt; 2
[2] =&gt; 2
[3] =&gt; 2
[4] =&gt; 2
Main threading exit.
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.5-Waiting-for-spawned-thread-exits">1.5 Waiting for spawned thread exits<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.5-Waiting-for-spawned-thread-exits">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [19]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># uses mutexes to know when threads are done in parent/main thread,</span>
<span class="c1"># instead of time.sleep; lock stdout to avoid comingled prints;</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [20]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">thread</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [21]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">stdoutmutex</span> <span class="o">=</span> <span class="n">thread</span><span class="o">.</span><span class="n">allocate_lock</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [22]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># exitmutex = [thread.allocate_lock() for i in range(5)]</span>
<span class="n">exitmutex</span> <span class="o">=</span> <span class="p">[</span><span class="kc">False</span><span class="p">]</span> <span class="o">*</span> <span class="mi">5</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [23]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">counter</span><span class="p">(</span><span class="n">myId</span><span class="p">,</span> <span class="n">count</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
        <span class="n">stdoutmutex</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
        <span class="nb">print</span> <span class="s1">'[</span><span class="si">%s</span><span class="s1">] =&gt; </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">myId</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
        <span class="n">stdoutmutex</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
<span class="c1">#     exitmutex[myId].acquire() # signal main thread</span>
    <span class="n">exitmutex</span><span class="p">[</span><span class="n">myId</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [41]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
    <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">counter</span><span class="p">,(</span><span class="n">i</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [42]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">for</span> <span class="n">mutex</span> <span class="ow">in</span> <span class="n">exitmutex</span><span class="p">:</span>
<span class="c1">#     while not mutex.locked():pass</span>
    <span class="k">while</span> <span class="ow">not</span> <span class="kc">True</span><span class="p">:</span><span class="k">pass</span>
<span class="nb">print</span> <span class="s1">'Main threading exit.'</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>[0] =&gt; 0
[1] =&gt; 0
[2] =&gt; 0
[3] =&gt; 0
[4] =&gt; 0
[0] =&gt; 1
[1] =&gt; 1
[2] =&gt; 1
[3] =&gt; 1
[4] =&gt; 1
[0] =&gt; 2
[1] =&gt; 2
[2] =&gt; 2
[4] =&gt; 2
[3] =&gt; 2
Main threading exit.
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="2.The-threading-Module">2.The threading Module<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.The-threading-Module">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="2.1-thread-classes">2.1 thread-classes<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.1-thread-classes">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [57]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">threading</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [66]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">class</span> <span class="nc">mythread</span><span class="p">(</span><span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">):</span>          <span class="c1"># subclass Thread object</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">myId</span><span class="p">,</span> <span class="n">count</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">myId</span> <span class="o">=</span> <span class="n">myId</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">count</span> <span class="o">=</span> <span class="n">count</span>
        <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
    <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">count</span><span class="p">):</span>
            <span class="n">stdoutmutex</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
            <span class="nb">print</span> <span class="s1">'[</span><span class="si">%s</span><span class="s1">] =&gt; </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">myId</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
            <span class="n">stdoutmutex</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [67]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">stdoutmutex</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span> <span class="c1"># same as thread.allocate_lock( )</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [68]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">threads</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
    <span class="n">thread</span> <span class="o">=</span> <span class="n">mythread</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
    <span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
    <span class="n">threads</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">thread</span><span class="p">)</span>
<span class="k">for</span> <span class="n">thread</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span>
    <span class="n">thread</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="nb">print</span> <span class="s1">'Main threading exit.'</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>[0] =&gt; 0
[0] =&gt; 1
[0] =&gt; 2
[1] =&gt; 0
[1] =&gt; 1
[1] =&gt; 2
[2] =&gt; 0
[2] =&gt; 1
[2] =&gt; 2
[3] =&gt; 0
[3] =&gt; 1
[3] =&gt; 2
[4] =&gt; 0
[4] =&gt; 1
[4] =&gt; 2
Main threading exit.
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="2.2-Other-ways-to-code-threads-with-threading">2.2 Other ways to code threads with threading<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.2-Other-ways-to-code-threads-with-threading">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [94]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">threading</span>
<span class="c1"># subclass with state</span>
<span class="k">class</span> <span class="nc">mythread</span><span class="p">(</span><span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">i</span> <span class="o">=</span> <span class="n">i</span>
        <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
    <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="nb">print</span> <span class="bp">self</span><span class="o">.</span><span class="n">i</span><span class="o">**</span><span class="mi">32</span>
<span class="n">mythread</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">start</span><span class="p">(</span> <span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>4294967296
4294967296
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [95]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">action</span><span class="p">(</span><span class="n">i</span><span class="p">):</span>
    <span class="nb">print</span> <span class="n">i</span><span class="o">**</span><span class="mi">32</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [96]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># pass action in</span>
<span class="n">thread</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="p">(</span><span class="k">lambda</span><span class="p">:</span> <span class="n">action</span><span class="p">(</span><span class="n">i</span><span class="p">)))</span>
<span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">(</span> <span class="p">)</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [98]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># same but no lambda wrapper for state</span>
<span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">action</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,))</span><span class="o">.</span><span class="n">start</span><span class="p">(</span> <span class="p">)</span>

<span class="c1"># basic thread module</span>
<span class="kn">import</span> <span class="nn">thread</span>
<span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">action</span><span class="p">,</span> <span class="p">(</span><span class="mi">2</span><span class="p">,))</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt output_prompt">Out[98]:</div>


<div class="output_text output_subarea output_execute_result">
<pre>140349501798144</pre>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [99]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># a non-thread class with state, OOP</span>
<span class="k">class</span> <span class="nc">Power</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">i</span> <span class="o">=</span> <span class="n">i</span>
    <span class="k">def</span> <span class="nf">action</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="nb">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">i</span> <span class="o">**</span> <span class="mi">32</span><span class="p">)</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">Power</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">obj</span><span class="o">.</span><span class="n">action</span><span class="p">)</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="c1"># thread runs bound method</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>4294967296
4294967296
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [101]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># nested scope to retain state</span>
<span class="k">def</span> <span class="nf">action</span><span class="p">(</span><span class="n">i</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">power</span><span class="p">():</span>
        <span class="nb">print</span><span class="p">(</span><span class="n">i</span> <span class="o">**</span> <span class="mi">32</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">power</span>
<span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">action</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="c1"># thread runs returned function</span>
<span class="c1"># both with basic thread module</span>
<span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">obj</span><span class="o">.</span><span class="n">action</span><span class="p">,</span> <span class="p">())</span>
<span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">action</span><span class="p">(</span><span class="mi">2</span><span class="p">),</span> <span class="p">())</span> <span class="c1"># thread runs a callable object</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt output_prompt">Out[101]:</div>


<div class="output_text output_subarea output_execute_result">
<pre>140349510190848</pre>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="2.3-Synchronizing-access-to-shared-objects-and-names-revisited">2.3 Synchronizing access to shared objects and names revisited<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.3-Synchronizing-access-to-shared-objects-and-names-revisited">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [128]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">threading</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">adder</span><span class="p">():</span>
    <span class="k">global</span> <span class="n">count</span>
    <span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c1">#     time.sleep(0.5)</span>
    <span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">threads</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
    <span class="n">thread</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">adder</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">())</span>
    <span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
    <span class="n">threads</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">thread</span><span class="p">)</span>
<span class="k">for</span> <span class="n">thread</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span> <span class="n">thread</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="nb">print</span> <span class="n">count</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>200
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [130]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#prints 200 each time, because shared resource access synchronized</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">adder</span><span class="p">(</span><span class="n">addlock</span><span class="p">):</span>
    <span class="k">global</span> <span class="n">count</span>
    <span class="k">with</span> <span class="n">addlock</span><span class="p">:</span>
        <span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c1">#     time.sleep(0.5)</span>
    <span class="k">with</span> <span class="n">addlock</span><span class="p">:</span>
        <span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">addlock</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span>
<span class="n">threads</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
    <span class="n">thread</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">adder</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">addlock</span><span class="p">,))</span>
    <span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
    <span class="n">threads</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">thread</span><span class="p">)</span>
<span class="k">for</span> <span class="n">thread</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span> <span class="n">thread</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="nb">print</span> <span class="n">count</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>200
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="The-queue-Module">The queue Module<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#The-queue-Module">¶</a>
</h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [131]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#producer and consumer threads communicating with a shared queue</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">numconsumers</span> <span class="o">=</span> <span class="mi">2</span> <span class="c1"># how many consumers to start</span>
<span class="n">numproducers</span> <span class="o">=</span> <span class="mi">4</span>  <span class="c1"># how many producers to start</span>
<span class="n">nummessages</span> <span class="o">=</span> <span class="mi">4</span>  <span class="c1"># messages per producer to put</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">thread</span><span class="o">,</span> <span class="nn">Queue</span><span class="o">,</span> <span class="nn">time</span>
<span class="n">safeprint</span> <span class="o">=</span> <span class="n">thread</span><span class="o">.</span><span class="n">allocate_lock</span><span class="p">()</span>  <span class="c1"># else print may overlap</span>
<span class="n">dataQueue</span> <span class="o">=</span> <span class="n">Queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>        <span class="c1"># shared global. infinite size</span>

<span class="k">def</span> <span class="nf">producer</span><span class="p">(</span><span class="n">idnum</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">msgnum</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nummessages</span><span class="p">):</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">idnum</span><span class="p">)</span>
        <span class="n">dataQueue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'[producer id=</span><span class="si">%d</span><span class="s1">, count=</span><span class="si">%d</span><span class="s1">]'</span> <span class="o">%</span> <span class="p">(</span><span class="n">idnum</span><span class="p">,</span> <span class="n">msgnum</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">idnum</span><span class="p">):</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
        <span class="k">try</span><span class="p">:</span>
            <span class="n">data</span> <span class="o">=</span> <span class="n">dataQueue</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">block</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
        <span class="k">except</span> <span class="n">Queue</span><span class="o">.</span><span class="n">Empty</span><span class="p">:</span>
            <span class="k">pass</span>
        <span class="k">else</span><span class="p">:</span>           
            <span class="n">safeprint</span><span class="o">.</span><span class="n">acquire</span><span class="p">(</span> <span class="p">)</span>
            <span class="nb">print</span> <span class="s1">'consumer'</span><span class="p">,</span> <span class="n">idnum</span><span class="p">,</span> <span class="s1">'got =&gt;'</span><span class="p">,</span> <span class="n">data</span>
            <span class="n">safeprint</span><span class="o">.</span><span class="n">release</span><span class="p">(</span> <span class="p">)</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">numconsumers</span><span class="p">):</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">consumer</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,))</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">numproducers</span><span class="p">):</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">producer</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,))</span>
    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(((</span><span class="n">numproducers</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">nummessages</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'Main thread exit.'</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>consumer 0 got =&gt; [producer id=0, count=0]
consumer 1 got =&gt; [producer id=0, count=1]
consumer 0 got =&gt; [producer id=0, count=2]
consumer 1 got =&gt; [producer id=0, count=3]
consumer 0 got =&gt; [producer id=1, count=0]
consumer 0 got =&gt; [producer id=2, count=0]
consumer 1 got =&gt; [producer id=1, count=1]
consumer 0 got =&gt; [producer id=3, count=0]
consumer 1 got =&gt; [producer id=1, count=2]
consumer 0 got =&gt; [producer id=2, count=1]
consumer 1 got =&gt; [producer id=1, count=3]
consumer 0 got =&gt; [producer id=2, count=2]
consumer 1 got =&gt; [producer id=3, count=1]
consumer 0 got =&gt; [producer id=2, count=3]
consumer 0 got =&gt; [producer id=3, count=2]
consumer 0 got =&gt; [producer id=3, count=3]
Main thread exit.
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [10]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#same as queuetest.py, by queue object pass in as argument, not global</span>
<span class="n">numconsumers</span> <span class="o">=</span> <span class="mi">2</span>                  <span class="c1"># how many consumers to start</span>
<span class="n">numproducers</span> <span class="o">=</span> <span class="mi">4</span>                  <span class="c1"># how many producers to start</span>
<span class="n">nummessages</span>  <span class="o">=</span> <span class="mi">4</span>                  <span class="c1"># messages per producer to put</span>

<span class="kn">import</span> <span class="nn">thread</span><span class="o">,</span> <span class="nn">Queue</span><span class="o">,</span> <span class="nn">time</span>
<span class="n">safeprint</span> <span class="o">=</span> <span class="n">thread</span><span class="o">.</span><span class="n">allocate_lock</span><span class="p">()</span>    <span class="c1"># else prints may overlap</span>
<span class="n">dataQueue</span> <span class="o">=</span> <span class="n">Queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>             <span class="c1"># shared global, infinite size</span>

<span class="k">def</span> <span class="nf">producer</span><span class="p">(</span><span class="n">idnum</span><span class="p">,</span> <span class="n">dataqueue</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">msgnum</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nummessages</span><span class="p">):</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">idnum</span><span class="p">)</span>
        <span class="n">dataqueue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'[producer id=</span><span class="si">%d</span><span class="s1">, count=</span><span class="si">%d</span><span class="s1">]'</span> <span class="o">%</span> <span class="p">(</span><span class="n">idnum</span><span class="p">,</span> <span class="n">msgnum</span><span class="p">))</span>

<span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">idnum</span><span class="p">,</span> <span class="n">dataqueue</span><span class="p">):</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
        <span class="k">try</span><span class="p">:</span>
            <span class="n">data</span> <span class="o">=</span> <span class="n">dataqueue</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">block</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
        <span class="k">except</span> <span class="n">Queue</span><span class="o">.</span><span class="n">Empty</span><span class="p">:</span>
            <span class="k">pass</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="k">with</span> <span class="n">safeprint</span><span class="p">:</span>
                <span class="nb">print</span> <span class="s1">'consumer'</span><span class="p">,</span> <span class="n">idnum</span><span class="p">,</span> <span class="s1">'got =&gt;'</span><span class="p">,</span> <span class="n">data</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">numconsumers</span><span class="p">):</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">consumer</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">dataQueue</span><span class="p">))</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">numproducers</span><span class="p">):</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">producer</span><span class="p">,</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">dataQueue</span><span class="p">))</span>
    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(((</span><span class="n">numproducers</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">nummessages</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
    <span class="nb">print</span> <span class="s1">'Main thread exit.'</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>consumer 1 got =&gt; [producer id=0, count=0]
consumer 0 got =&gt; [producer id=0, count=1]
consumer 1 got =&gt; [producer id=0, count=2]
consumer 0 got =&gt; [producer id=0, count=3]
consumer 0 got =&gt; [producer id=1, count=0]
consumer 1 got =&gt; [producer id=2, count=0]
consumer 0 got =&gt; [producer id=1, count=1]
consumer 1 got =&gt; [producer id=3, count=0]
consumer 0 got =&gt; [producer id=1, count=2]
consumer 1 got =&gt; [producer id=2, count=1]
consumer 1 got =&gt; [producer id=1, count=3]
consumer 1 got =&gt; [producer id=3, count=1]
consumer 1 got =&gt; [producer id=2, count=2]
consumer 1 got =&gt; [producer id=2, count=3]
consumer 1 got =&gt; [producer id=3, count=2]
consumer 1 got =&gt; [producer id=3, count=3]
Main thread exit.
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [12]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#same as queuetest2.py, but uses threading, not _threads</span>
<span class="n">numconsumers</span> <span class="o">=</span> <span class="mi">2</span>                  <span class="c1"># how many consumers to start</span>
<span class="n">numproducers</span> <span class="o">=</span> <span class="mi">4</span>                  <span class="c1"># how many producers to start</span>
<span class="n">nummessages</span>  <span class="o">=</span> <span class="mi">4</span>                  <span class="c1"># messages per producer to put</span>

<span class="kn">import</span> <span class="nn">threading</span><span class="o">,</span> <span class="nn">Queue</span><span class="o">,</span> <span class="nn">time</span>
<span class="n">safeprint</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span>          <span class="c1"># else prints may overlap</span>
<span class="n">dataQueue</span> <span class="o">=</span> <span class="n">Queue</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>             <span class="c1"># shared global, infinite size</span>

<span class="k">def</span> <span class="nf">producer</span><span class="p">(</span><span class="n">idnum</span><span class="p">,</span> <span class="n">dataqueue</span><span class="p">):</span>
    <span class="k">for</span> <span class="n">msgnum</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nummessages</span><span class="p">):</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">idnum</span><span class="p">)</span>
        <span class="n">dataqueue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="s1">'[producer id=</span><span class="si">%d</span><span class="s1">, count=</span><span class="si">%d</span><span class="s1">]'</span> <span class="o">%</span> <span class="p">(</span><span class="n">idnum</span><span class="p">,</span> <span class="n">msgnum</span><span class="p">))</span>

<span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">idnum</span><span class="p">,</span> <span class="n">dataqueue</span><span class="p">):</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
        <span class="k">try</span><span class="p">:</span>
            <span class="n">data</span> <span class="o">=</span> <span class="n">dataqueue</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">block</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
        <span class="k">except</span> <span class="n">Queue</span><span class="o">.</span><span class="n">Empty</span><span class="p">:</span>
            <span class="k">pass</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="k">with</span> <span class="n">safeprint</span><span class="p">:</span>
                <span class="nb">print</span> <span class="s1">'consumer'</span><span class="p">,</span> <span class="n">idnum</span><span class="p">,</span> <span class="s1">'got =&gt;'</span><span class="p">,</span> <span class="n">data</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">numconsumers</span><span class="p">):</span>
        <span class="n">thread</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">consumer</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">dataQueue</span><span class="p">))</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="kc">True</span>  <span class="c1"># else cannot exit!</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>

    <span class="n">waitfor</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">numproducers</span><span class="p">):</span>
        <span class="n">thread</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">producer</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">dataQueue</span><span class="p">))</span>
        <span class="n">waitfor</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">thread</span><span class="p">)</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>

    <span class="k">for</span> <span class="n">thread</span> <span class="ow">in</span> <span class="n">waitfor</span><span class="p">:</span> <span class="n">thread</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>    <span class="c1"># or time.sleep() long enough here</span>
    <span class="nb">print</span> <span class="s1">'Main thread exit.'</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>consumer 0 got =&gt; [producer id=3, count=3]
consumer 1 got =&gt; [producer id=0, count=0]
consumer 0 got =&gt; [producer id=0, count=1]
consumer 0 got =&gt; [producer id=0, count=2]
consumer 1 got =&gt; [producer id=0, count=3]
consumer 1 got =&gt; [producer id=1, count=0]
consumer 0 got =&gt; [producer id=1, count=1]
consumer 1 got =&gt; [producer id=2, count=0]
consumer 0 got =&gt; [producer id=1, count=2]
consumer 1 got =&gt; [producer id=3, count=0]
consumer 0 got =&gt; [producer id=1, count=3]
consumer 0 got =&gt; [producer id=2, count=1]
consumer 0 got =&gt; [producer id=2, count=2]
consumer 0 got =&gt; [producer id=3, count=1]
consumer 0 got =&gt; [producer id=2, count=3]
consumer 0 got =&gt; [producer id=3, count=2]
Main thread exit.
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Program-Exits">Program Exits<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#Program-Exits">¶</a>
</h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="1.Process-Exit-Status-and-Shared-State">1.Process Exit Status and Shared State<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.Process-Exit-Status-and-Shared-State">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">exitstat</span> <span class="o">=</span> <span class="mi">0</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">child</span><span class="p">():</span>
    <span class="k">global</span> <span class="n">exitstat</span>
    <span class="n">exitstat</span> <span class="o">+=</span> <span class="mi">1</span>
    <span class="nb">print</span> <span class="s1">'Hello from child'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">exitstat</span>
    <span class="n">os</span><span class="o">.</span><span class="n">_exit</span><span class="p">(</span><span class="n">exitstat</span><span class="p">)</span>
    <span class="nb">print</span> <span class="s1">'never reached'</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [12]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">parent</span><span class="p">():</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">newpid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span>
        <span class="k">if</span> <span class="n">newpid</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
            <span class="n">child</span><span class="p">()</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="n">pid</span><span class="p">,</span> <span class="n">status</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
            <span class="nb">print</span> <span class="s1">'Parent got'</span><span class="p">,</span> <span class="n">pid</span><span class="p">,</span> <span class="n">status</span><span class="p">,</span> <span class="p">(</span><span class="n">status</span> <span class="o">&gt;&gt;</span> <span class="mi">8</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">raw_input</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'q'</span><span class="p">:</span> <span class="k">break</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [13]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">parent</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Parent got 4367 256 1
Hello from child 4367 1

Parent got 4373 256 1
Hello from child 4373 1

Parent got 4379 256 1
Hello from child 4379 1

Parent got 4385 256 1
Hello from child 4385 1

Parent got 4391 256 1
Hello from child 4391 1
q
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="2.-Thread-Exits-and-Shared-State">2. Thread Exits and Shared State<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.-Thread-Exits-and-Shared-State">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [20]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">thread</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [21]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">exitstat</span> <span class="o">=</span> <span class="mi">0</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [22]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">child</span><span class="p">():</span>
    <span class="k">global</span> <span class="n">exitstat</span>
    <span class="n">exitstat</span> <span class="o">+=</span> <span class="mi">1</span>
    <span class="n">threadid</span> <span class="o">=</span> <span class="n">thread</span><span class="o">.</span><span class="n">get_ident</span><span class="p">()</span>
    <span class="nb">print</span> <span class="s1">'Hello from child'</span><span class="p">,</span> <span class="n">threadid</span><span class="p">,</span> <span class="n">exitstat</span>
    <span class="n">thread</span><span class="o">.</span><span class="n">exit</span><span class="p">()</span>
    <span class="nb">print</span> <span class="s1">'never reached'</span>
<span class="k">def</span> <span class="nf">parent</span><span class="p">():</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">thread</span><span class="o">.</span><span class="n">start_new_thread</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="p">())</span>
        <span class="k">if</span> <span class="n">raw_input</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'q'</span><span class="p">:</span> <span class="k">break</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span> <span class="n">parent</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>
Hello from child 140613521495808 1

Hello from child 140613521495808 2
q
Hello from child 140613521495808 3
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># As we’ve learned, a thread normally exits silently when the function it runs returns,</span>
<span class="c1"># and the function return value is ignored. Optionally, the _thread.exit function can be</span>
<span class="c1"># called to terminate the calling thread explicitly and silently. This call works almost</span>
<span class="c1"># exactly like sys.exit (but takes no return status argument), and it works by raising a</span>
<span class="c1"># SystemExit exception in the calling thread. Because of that, a thread can also prema-</span>
<span class="c1"># turely end by calling sys.exit or by directly raising SystemExit . Be sure not to call</span>
<span class="c1"># os._exit within a thread function, though—doing so can have odd results (the last time</span>
<span class="c1"># I tried, it hung the entire process on my Linux system and killed every thread in the</span>
<span class="c1"># process on Windows!).</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># keep in mind that threads and processes have default lifespan models,</span>
<span class="c1"># which we explored earlier. By way of review, when child threads are still running, the</span>
<span class="c1"># two thread modules’ behavior differs—programs on most platforms exit when the pa-</span>
<span class="c1"># rent thread does under _thread , but not normally under threading unless children are</span>
<span class="c1"># made daemons. When using processes, children normally outlive their parent. This</span>
<span class="c1"># different process behavior makes sense if you remember that threads are in-process</span>
<span class="c1"># function calls, but processes are more independent and autonomous.</span>
<span class="c1"># When used well, exit status can be used to implement error detection and simple com-</span>
<span class="c1"># munication protocols in systems composed of command-line scripts. But having said</span>
<span class="c1"># that, I should underscore that most scripts do simply fall off the end of the source to</span>
<span class="c1"># exit, and most thread functions simply return; explicit exit calls are generally employed</span>
<span class="c1"># for exceptional conditions and in limited contexts only.</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Interprocess-Communication">Interprocess Communication<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#Interprocess-Communication">¶</a>
</h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="1.-Anonymous-pipe">1. Anonymous pipe<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.-Anonymous-pipe">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.1-Anonymous-pipe-basics">1.1 Anonymous pipe basics<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.1-Anonymous-pipe-basics">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [28]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [29]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">child</span><span class="p">(</span><span class="n">pipeout</span><span class="p">):</span>
    <span class="n">zzz</span> <span class="o">=</span> <span class="mi">0</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">zzz</span><span class="p">)</span>                                    <span class="c1">#make parent wait</span>
        <span class="n">msg</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Spam </span><span class="si">%03d</span><span class="s1">'</span> <span class="o">%</span> <span class="n">zzz</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span> <span class="c1">#pipes are binary bytes</span>
        <span class="n">os</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">pipeout</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span>                       <span class="c1">#send to parent</span>
        <span class="n">zzz</span> <span class="o">=</span> <span class="p">(</span><span class="n">zzz</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="mi">5</span>                                <span class="c1">#goto 0 after 4</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [32]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">parent</span><span class="p">():</span>
    <span class="n">pipein</span><span class="p">,</span><span class="n">pipeout</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">pipe</span><span class="p">()</span>
    <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
        <span class="n">child</span><span class="p">(</span><span class="n">pipeout</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
            <span class="n">line</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">pipein</span><span class="p">,</span> <span class="mi">32</span><span class="p">)</span>
            <span class="c1"># blocks until data sent</span>
            <span class="nb">print</span> <span class="s1">'Parent </span><span class="si">%d</span><span class="s1"> got [</span><span class="si">%s</span><span class="s1">] at </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">line</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">())</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [34]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">parent</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.2-Wrapping-pipe-descriptors-in-file-objects">1.2 Wrapping pipe descriptors in file objects<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.2-Wrapping-pipe-descriptors-in-file-objects">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [39]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># same as pipe1.py, but wrap pipe input in stdio file object</span>
<span class="c1"># to read by line, and close unused pipe fds in both processes</span>
<span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span>
<span class="k">def</span> <span class="nf">child</span><span class="p">(</span><span class="n">pipeout</span><span class="p">):</span>
    <span class="n">zzz</span> <span class="o">=</span> <span class="mi">0</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">zzz</span><span class="p">)</span>
        <span class="n">msg</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Spam </span><span class="si">%03d</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="n">zzz</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span>
        <span class="n">os</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">pipeout</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span>
        <span class="n">zzz</span> <span class="o">=</span> <span class="p">(</span><span class="n">zzz</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="mi">5</span>
<span class="k">def</span> <span class="nf">parent</span><span class="p">():</span>
    <span class="n">pipein</span><span class="p">,</span> <span class="n">pipeout</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">pipe</span><span class="p">()</span>
    <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
        <span class="n">os</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">pipein</span><span class="p">)</span>
        <span class="n">child</span><span class="p">(</span><span class="n">pipeout</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">os</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">pipeout</span><span class="p">)</span>
        <span class="n">pipein</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fdopen</span><span class="p">(</span><span class="n">pipein</span><span class="p">)</span>
        <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
            <span class="n">line</span> <span class="o">=</span> <span class="n">pipein</span><span class="o">.</span><span class="n">readline</span><span class="p">()[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
            <span class="nb">print</span><span class="p">(</span><span class="s1">'Parent </span><span class="si">%d</span><span class="s1"> got [</span><span class="si">%s</span><span class="s1">] at </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">line</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()))</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [38]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">parent</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.1.3-Anonymous-pipes-and-threads">1.1.3 Anonymous pipes and threads<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.1.3-Anonymous-pipes-and-threads">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">threading</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">child</span><span class="p">(</span><span class="n">pipeout</span><span class="p">):</span>
    <span class="n">zzz</span> <span class="o">=</span> <span class="mi">0</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">zzz</span><span class="p">)</span>
        <span class="n">msg</span> <span class="o">=</span> <span class="p">(</span><span class="s2">"Spam </span><span class="si">%03d</span><span class="s2">"</span> <span class="o">%</span> <span class="n">zzz</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span>
        <span class="n">os</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">pipeout</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span>
        <span class="n">zzz</span> <span class="o">=</span> <span class="p">(</span><span class="n">zzz</span> <span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="mi">5</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">parent</span><span class="p">(</span><span class="n">pipein</span><span class="p">):</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">line</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">pipein</span><span class="p">,</span> <span class="mi">32</span><span class="p">)</span>
        <span class="nb">print</span> <span class="s1">'Parent </span><span class="si">%d</span><span class="s1"> got [</span><span class="si">%s</span><span class="s1">] at </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">line</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">())</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [4]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">pipein</span> <span class="p">,</span><span class="n">pipeout</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">pipe</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [5]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">child</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">pipeout</span><span class="p">,))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">parent</span><span class="p">(</span><span class="n">pipein</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Parent 3257 got [Spam 003Spam 004Spam 000Spam 001] at 1400549465.95
Parent 3257 got [Spam 002Spam 003Spam 004Spam 000] at 1400549465.95
Parent 3257 got [Spam 001Spam 002Spam 003] at 1400549465.95
Parent 3257 got [Spam 004] at 1400549468.75
Parent 3257 got [Spam 000] at 1400549468.76
Parent 3257 got [Spam 001] at 1400549469.76
Parent 3257 got [Spam 002] at 1400549471.76
Parent 3257 got [Spam 003] at 1400549474.76
</pre>
</div>
</div>

<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansi-red-intense-fg ansi-bold">---------------------------------------------------------------------------</span>
<span class="ansi-red-intense-fg ansi-bold">KeyboardInterrupt</span>                         Traceback (most recent call last)
<span class="ansi-green-intense-fg ansi-bold">&lt;ipython-input-7-a47c722744b8&gt;</span> in <span class="ansi-cyan-fg">&lt;module&gt;</span><span class="ansi-blue-intense-fg ansi-bold">()</span>
<span class="ansi-green-intense-fg ansi-bold">----&gt; 1</span><span class="ansi-yellow-intense-fg ansi-bold"> </span>parent<span class="ansi-yellow-intense-fg ansi-bold">(</span>pipein<span class="ansi-yellow-intense-fg ansi-bold">)</span>

<span class="ansi-green-intense-fg ansi-bold">&lt;ipython-input-3-218f40ee8ea0&gt;</span> in <span class="ansi-cyan-fg">parent</span><span class="ansi-blue-intense-fg ansi-bold">(pipein)</span>
<span class="ansi-green-fg">      1</span> <span class="ansi-green-intense-fg ansi-bold">def</span> parent<span class="ansi-yellow-intense-fg ansi-bold">(</span>pipein<span class="ansi-yellow-intense-fg ansi-bold">)</span><span class="ansi-yellow-intense-fg ansi-bold">:</span>
<span class="ansi-green-fg">      2</span>     <span class="ansi-green-intense-fg ansi-bold">while</span> True<span class="ansi-yellow-intense-fg ansi-bold">:</span>
<span class="ansi-green-intense-fg ansi-bold">----&gt; 3</span><span class="ansi-yellow-intense-fg ansi-bold">         </span>line <span class="ansi-yellow-intense-fg ansi-bold">=</span> os<span class="ansi-yellow-intense-fg ansi-bold">.</span>read<span class="ansi-yellow-intense-fg ansi-bold">(</span>pipein<span class="ansi-yellow-intense-fg ansi-bold">,</span> <span class="ansi-cyan-intense-fg ansi-bold">32</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>
<span class="ansi-green-fg">      4</span>         <span class="ansi-green-intense-fg ansi-bold">print</span> <span class="ansi-blue-intense-fg ansi-bold">'Parent %d got [%s] at %s'</span> <span class="ansi-yellow-intense-fg ansi-bold">%</span> <span class="ansi-yellow-intense-fg ansi-bold">(</span>os<span class="ansi-yellow-intense-fg ansi-bold">.</span>getpid<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-yellow-intense-fg ansi-bold">)</span><span class="ansi-yellow-intense-fg ansi-bold">,</span> line<span class="ansi-yellow-intense-fg ansi-bold">,</span> time<span class="ansi-yellow-intense-fg ansi-bold">.</span>time<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-yellow-intense-fg ansi-bold">)</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>

<span class="ansi-red-intense-fg ansi-bold">KeyboardInterrupt</span>: </pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.1.4-Bidirectional-IPC-with-anonymous-pipes">1.1.4 Bidirectional IPC with anonymous pipes<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.1.4-Bidirectional-IPC-with-anonymous-pipes">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [9]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># spawn a child process/program, connect my stdin/stdout to child process's</span>
<span class="c1"># stdout/stdin--my reads and writes map to output and input streams of the</span>
<span class="c1"># spawned program; much like tying together streams with subprocess module;</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [17]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">os</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [19]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">spawn</span><span class="p">(</span><span class="n">prog</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>                       <span class="c1"># pass progname, cmdline args</span>
    <span class="n">stdinFd</span>  <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">fileno</span><span class="p">()</span>             <span class="c1"># get descriptors for streams</span>
    <span class="n">stdoutFd</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">fileno</span><span class="p">()</span>            <span class="c1"># normally stdin=0, stdout=1</span>

    <span class="n">parentStdin</span><span class="p">,</span> <span class="n">childStdout</span>  <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">pipe</span><span class="p">()</span>     <span class="c1"># make two IPC pipe channels</span>
    <span class="n">childStdin</span><span class="p">,</span>  <span class="n">parentStdout</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">pipe</span><span class="p">()</span>     <span class="c1"># pipe returns (inputfd, outoutfd)</span>
    <span class="n">pid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span>                           <span class="c1"># make a copy of this process</span>
    <span class="k">if</span> <span class="n">pid</span><span class="p">:</span>
        <span class="n">os</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">childStdout</span><span class="p">)</span>                 <span class="c1"># in parent process after fork:</span>
        <span class="n">os</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">childStdin</span><span class="p">)</span>                  <span class="c1"># close child ends in parent</span>
        <span class="n">os</span><span class="o">.</span><span class="n">dup2</span><span class="p">(</span><span class="n">parentStdin</span><span class="p">,</span>  <span class="n">stdinFd</span><span class="p">)</span>        <span class="c1"># my sys.stdin copy  = pipe1[0]</span>
        <span class="n">os</span><span class="o">.</span><span class="n">dup2</span><span class="p">(</span><span class="n">parentStdout</span><span class="p">,</span> <span class="n">stdoutFd</span><span class="p">)</span>       <span class="c1"># my sys.stdout copy = pipe2[1]</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">os</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">parentStdin</span><span class="p">)</span>                 <span class="c1"># in child process after fork:</span>
        <span class="n">os</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">parentStdout</span><span class="p">)</span>                <span class="c1"># close parent ends in child</span>
        <span class="n">os</span><span class="o">.</span><span class="n">dup2</span><span class="p">(</span><span class="n">childStdin</span><span class="p">,</span>  <span class="n">stdinFd</span><span class="p">)</span>         <span class="c1"># my sys.stdin copy  = pipe2[0]</span>
        <span class="n">os</span><span class="o">.</span><span class="n">dup2</span><span class="p">(</span><span class="n">childStdout</span><span class="p">,</span> <span class="n">stdoutFd</span><span class="p">)</span>        <span class="c1"># my sys.stdout copy = pipe1[1]</span>
        <span class="n">args</span> <span class="o">=</span> <span class="p">(</span><span class="n">prog</span><span class="p">,)</span> <span class="o">+</span> <span class="n">args</span>
        <span class="n">os</span><span class="o">.</span><span class="n">execvp</span><span class="p">(</span><span class="n">prog</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span>                 <span class="c1"># new program in this process</span>
        <span class="k">assert</span> <span class="kc">False</span><span class="p">,</span> <span class="s1">'execvp failed!'</span>        <span class="c1"># os.exec call never returns here</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="n">mypid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">()</span>
    <span class="n">spawn</span><span class="p">(</span><span class="s1">'python'</span><span class="p">,</span> <span class="s1">'-u'</span><span class="p">,</span> <span class="s1">'pipes-testchild.py'</span><span class="p">,</span> <span class="s1">'spam'</span><span class="p">)</span>     <span class="c1"># fork child program</span>

    <span class="nb">print</span><span class="p">(</span><span class="s1">'Hello 1 from parent'</span><span class="p">,</span> <span class="n">mypid</span><span class="p">)</span>               <span class="c1"># to child's stdin</span>
    <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>                                <span class="c1"># subvert stdio buffering</span>
    <span class="n">reply</span> <span class="o">=</span> <span class="nb">input</span><span class="p">()</span>                                   <span class="c1"># from child's stdout</span>
    <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'Parent got: "</span><span class="si">%s</span><span class="s1">"</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="n">reply</span><span class="p">)</span>    <span class="c1"># stderr not tied to pipe!</span>

    <span class="nb">print</span><span class="p">(</span><span class="s1">'Hello 2 from parent'</span><span class="p">,</span> <span class="n">mypid</span><span class="p">)</span>
    <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
    <span class="n">reply</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
    <span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'Parent got: "</span><span class="si">%s</span><span class="s1">"</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="n">reply</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansi-red-intense-fg ansi-bold">---------------------------------------------------------------------------</span>
<span class="ansi-red-intense-fg ansi-bold">UnsupportedOperation</span>                      Traceback (most recent call last)
<span class="ansi-green-intense-fg ansi-bold">&lt;ipython-input-19-8b7123926819&gt;</span> in <span class="ansi-cyan-fg">&lt;module&gt;</span><span class="ansi-blue-intense-fg ansi-bold">()</span>
<span class="ansi-green-fg">     22</span> <span class="ansi-green-intense-fg ansi-bold">if</span> __name__ <span class="ansi-yellow-intense-fg ansi-bold">==</span> <span class="ansi-blue-intense-fg ansi-bold">'__main__'</span><span class="ansi-yellow-intense-fg ansi-bold">:</span>
<span class="ansi-green-fg">     23</span>     mypid <span class="ansi-yellow-intense-fg ansi-bold">=</span> os<span class="ansi-yellow-intense-fg ansi-bold">.</span>getpid<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>
<span class="ansi-green-intense-fg ansi-bold">---&gt; 24</span><span class="ansi-yellow-intense-fg ansi-bold">     </span>spawn<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-blue-intense-fg ansi-bold">'python'</span><span class="ansi-yellow-intense-fg ansi-bold">,</span> <span class="ansi-blue-intense-fg ansi-bold">'-u'</span><span class="ansi-yellow-intense-fg ansi-bold">,</span> <span class="ansi-blue-intense-fg ansi-bold">'pipes-testchild.py'</span><span class="ansi-yellow-intense-fg ansi-bold">,</span> <span class="ansi-blue-intense-fg ansi-bold">'spam'</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>     <span class="ansi-red-intense-fg ansi-bold"># fork child program</span>
<span class="ansi-green-fg">     25</span> 
<span class="ansi-green-fg">     26</span>     <span class="ansi-green-intense-fg ansi-bold">print</span><span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-blue-intense-fg ansi-bold">'Hello 1 from parent'</span><span class="ansi-yellow-intense-fg ansi-bold">,</span> mypid<span class="ansi-yellow-intense-fg ansi-bold">)</span>               <span class="ansi-red-intense-fg ansi-bold"># to child's stdin</span>

<span class="ansi-green-intense-fg ansi-bold">&lt;ipython-input-19-8b7123926819&gt;</span> in <span class="ansi-cyan-fg">spawn</span><span class="ansi-blue-intense-fg ansi-bold">(prog, *args)</span>
<span class="ansi-green-fg">      1</span> <span class="ansi-green-intense-fg ansi-bold">def</span> spawn<span class="ansi-yellow-intense-fg ansi-bold">(</span>prog<span class="ansi-yellow-intense-fg ansi-bold">,</span> <span class="ansi-yellow-intense-fg ansi-bold">*</span>args<span class="ansi-yellow-intense-fg ansi-bold">)</span><span class="ansi-yellow-intense-fg ansi-bold">:</span>                       <span class="ansi-red-intense-fg ansi-bold"># pass progname, cmdline args</span>
<span class="ansi-green-fg">      2</span>     stdinFd  <span class="ansi-yellow-intense-fg ansi-bold">=</span> sys<span class="ansi-yellow-intense-fg ansi-bold">.</span>stdin<span class="ansi-yellow-intense-fg ansi-bold">.</span>fileno<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>             <span class="ansi-red-intense-fg ansi-bold"># get descriptors for streams</span>
<span class="ansi-green-intense-fg ansi-bold">----&gt; 3</span><span class="ansi-yellow-intense-fg ansi-bold">     </span>stdoutFd <span class="ansi-yellow-intense-fg ansi-bold">=</span> sys<span class="ansi-yellow-intense-fg ansi-bold">.</span>stdout<span class="ansi-yellow-intense-fg ansi-bold">.</span>fileno<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>            <span class="ansi-red-intense-fg ansi-bold"># normally stdin=0, stdout=1</span>
<span class="ansi-green-fg">      4</span> 
<span class="ansi-green-fg">      5</span>     parentStdin<span class="ansi-yellow-intense-fg ansi-bold">,</span> childStdout  <span class="ansi-yellow-intense-fg ansi-bold">=</span> os<span class="ansi-yellow-intense-fg ansi-bold">.</span>pipe<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>     <span class="ansi-red-intense-fg ansi-bold"># make two IPC pipe channels</span>

<span class="ansi-green-intense-fg ansi-bold">/usr/local/lib/python2.7/dist-packages/IPython/kernel/zmq/iostream.pyc</span> in <span class="ansi-cyan-fg">fileno</span><span class="ansi-blue-intense-fg ansi-bold">(self)</span>
<span class="ansi-green-fg">    185</span> 
<span class="ansi-green-fg">    186</span>     <span class="ansi-green-intense-fg ansi-bold">def</span> fileno<span class="ansi-yellow-intense-fg ansi-bold">(</span>self<span class="ansi-yellow-intense-fg ansi-bold">)</span><span class="ansi-yellow-intense-fg ansi-bold">:</span>
<span class="ansi-green-intense-fg ansi-bold">--&gt; 187</span><span class="ansi-yellow-intense-fg ansi-bold">         </span><span class="ansi-green-intense-fg ansi-bold">raise</span> UnsupportedOperation<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-blue-intense-fg ansi-bold">"IOStream has no fileno."</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>
<span class="ansi-green-fg">    188</span> 
<span class="ansi-green-fg">    189</span>     <span class="ansi-green-intense-fg ansi-bold">def</span> write<span class="ansi-yellow-intense-fg ansi-bold">(</span>self<span class="ansi-yellow-intense-fg ansi-bold">,</span> string<span class="ansi-yellow-intense-fg ansi-bold">)</span><span class="ansi-yellow-intense-fg ansi-bold">:</span>

<span class="ansi-red-intense-fg ansi-bold">UnsupportedOperation</span>: IOStream has no fileno.</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [14]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">sys</span>
<span class="n">mypid</span>     <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">()</span>
<span class="n">parentpid</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getppid</span><span class="p">()</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'Child </span><span class="si">%d</span><span class="s1"> of </span><span class="si">%d</span><span class="s1"> got arg: "</span><span class="si">%s</span><span class="s1">"</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">mypid</span><span class="p">,</span> <span class="n">parentpid</span><span class="p">,</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">):</span>
    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>              <span class="c1"># make parent process wait by sleeping here</span>
    <span class="n">recv</span> <span class="o">=</span> <span class="n">raw_input</span><span class="p">()</span>             <span class="c1"># stdin tied to pipe: comes from parent's stdout</span>
    <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
    <span class="n">send</span> <span class="o">=</span> <span class="s1">'Child </span><span class="si">%d</span><span class="s1"> got: [</span><span class="si">%s</span><span class="s1">]'</span> <span class="o">%</span> <span class="p">(</span><span class="n">mypid</span><span class="p">,</span> <span class="n">recv</span><span class="p">)</span>
    <span class="nb">print</span> <span class="n">send</span>                <span class="c1"># stdout tied to pipe: goes to parent's stdin</span>
    <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>         <span class="c1"># make sure it's sent now or else process blocks</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stderr output_text">
<pre>Child 3257 of 2729 got arg: "-f"
</pre>
</div>
</div>

<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>ss
Child 3257 got: [ss]
rt
Child 3257 got: [rt]
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="1.1.5-Output-stream-buffering-revisited:-Deadlocks-and-flushes">1.1.5 Output stream buffering revisited: Deadlocks and flushes<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1.1.5-Output-stream-buffering-revisited:-Deadlocks-and-flushes">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [20]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># Deadlock in general, though, is a bigger problem than we have space to address fully</span>
<span class="c1"># here. On the other hand, if you know enough that you want to do IPC in Python, you’re</span>
<span class="c1"># probably already a veteran of the deadlock wars.</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="2-Named-Pipes-(Fifos)">2 Named Pipes (Fifos)<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2-Named-Pipes-(Fifos)">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [23]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># named pipes; os.mkfifo is not available on Windows (without Cygwin); </span>
<span class="c1"># there is no reason to fork here, since fifo file pipes are external </span>
<span class="c1"># to processes--shared fds in parent/child processes are irrelevent;</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [24]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">sys</span>
<span class="n">fifoname</span> <span class="o">=</span> <span class="s1">'/tmp/pipefifo'</span>                       <span class="c1"># must open same name</span>

<span class="k">def</span> <span class="nf">child</span><span class="p">():</span>
    <span class="n">pipeout</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">fifoname</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">O_WRONLY</span><span class="p">)</span>     <span class="c1"># open fifo pipe file as fd</span>
    <span class="n">zzz</span> <span class="o">=</span> <span class="mi">0</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">zzz</span><span class="p">)</span>
        <span class="n">msg</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Spam </span><span class="si">%03d</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="n">zzz</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span>     <span class="c1"># binary as opened here</span>
        <span class="n">os</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">pipeout</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span>
        <span class="n">zzz</span> <span class="o">=</span> <span class="p">(</span><span class="n">zzz</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="o">%</span> <span class="mi">5</span>

<span class="k">def</span> <span class="nf">parent</span><span class="p">():</span>
    <span class="n">pipein</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">fifoname</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span>                 <span class="c1"># open fifo as text file object</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">line</span> <span class="o">=</span> <span class="n">pipein</span><span class="o">.</span><span class="n">readline</span><span class="p">()[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>            <span class="c1"># blocks until data sent</span>
        <span class="nb">print</span><span class="p">(</span><span class="s1">'Parent </span><span class="si">%d</span><span class="s1"> got "</span><span class="si">%s</span><span class="s1">" at </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">line</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()))</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [26]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">fifoname</span><span class="p">):</span>
        <span class="n">os</span><span class="o">.</span><span class="n">mkfifo</span><span class="p">(</span><span class="n">fifoname</span><span class="p">)</span>                      <span class="c1"># create a named pipe file</span>
    <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
        <span class="n">parent</span><span class="p">()</span>                                 <span class="c1"># run as parent if no args</span>
    <span class="k">else</span><span class="p">:</span>                                        <span class="c1"># else run as child process</span>
        <span class="n">child</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="3-Sockets">3 Sockets<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#3-Sockets">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [ ]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># sockets for cross-task communication: start threads to communicate over sockets;</span>
<span class="c1"># independent programs can too, because sockets are system-wide, much like fifos;</span>
<span class="c1"># see the GUI and Internet parts of the book for more realistic socket use cases;</span>
<span class="c1"># some socket servers may also need to talk to clients in threads or processes;</span>
<span class="c1"># sockets pass byte strings, but can be pickled objects or encoded Unicode text;</span>
<span class="c1"># caveat: prints in threads may need to be synchronized if their output overlaps;</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [27]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">socket</span> <span class="k">import</span> <span class="n">socket</span><span class="p">,</span> <span class="n">AF_INET</span><span class="p">,</span> <span class="n">SOCK_STREAM</span>     <span class="c1"># portable socket api</span>

<span class="n">port</span> <span class="o">=</span> <span class="mi">50008</span>                 <span class="c1"># port number identifies socket on machine</span>
<span class="n">host</span> <span class="o">=</span> <span class="s1">'localhost'</span>           <span class="c1"># server and client run on same local machine here</span>

<span class="k">def</span> <span class="nf">server</span><span class="p">():</span>
    <span class="n">sock</span> <span class="o">=</span> <span class="n">socket</span><span class="p">(</span><span class="n">AF_INET</span><span class="p">,</span> <span class="n">SOCK_STREAM</span><span class="p">)</span>         <span class="c1"># ip addresses tcp connection</span>
    <span class="n">sock</span><span class="o">.</span><span class="n">bind</span><span class="p">((</span><span class="s1">''</span><span class="p">,</span> <span class="n">port</span><span class="p">))</span>                       <span class="c1"># bind to port on this machine</span>
    <span class="n">sock</span><span class="o">.</span><span class="n">listen</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>                              <span class="c1"># allow up to 5 pending clients</span>
    <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
        <span class="n">conn</span><span class="p">,</span> <span class="n">addr</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">accept</span><span class="p">()</span>              <span class="c1"># wait for client to connect</span>
        <span class="n">data</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">recv</span><span class="p">(</span><span class="mi">1024</span><span class="p">)</span>                  <span class="c1"># read bytes data from this client</span>
        <span class="n">reply</span> <span class="o">=</span> <span class="s1">'server got: [</span><span class="si">%s</span><span class="s1">]'</span> <span class="o">%</span> <span class="n">data</span>       <span class="c1"># conn is a new connected socket</span>
        <span class="n">conn</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">reply</span><span class="o">.</span><span class="n">encode</span><span class="p">())</span>               <span class="c1"># send bytes reply back to client</span>

<span class="k">def</span> <span class="nf">client</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
    <span class="n">sock</span> <span class="o">=</span> <span class="n">socket</span><span class="p">(</span><span class="n">AF_INET</span><span class="p">,</span> <span class="n">SOCK_STREAM</span><span class="p">)</span>
    <span class="n">sock</span><span class="o">.</span><span class="n">connect</span><span class="p">((</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">))</span>                  <span class="c1"># connect to a socket port</span>
    <span class="n">sock</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">name</span><span class="o">.</span><span class="n">encode</span><span class="p">())</span>                    <span class="c1"># send bytes data to listener</span>
    <span class="n">reply</span> <span class="o">=</span> <span class="n">sock</span><span class="o">.</span><span class="n">recv</span><span class="p">(</span><span class="mi">1024</span><span class="p">)</span>                     <span class="c1"># receive bytes data from listener</span>
    <span class="n">sock</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>                                <span class="c1"># up to 1024 bytes in message</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'client got: [</span><span class="si">%s</span><span class="s1">]'</span> <span class="o">%</span> <span class="n">reply</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [29]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="kn">from</span> <span class="nn">threading</span> <span class="k">import</span> <span class="n">Thread</span>
    <span class="n">sthread</span> <span class="o">=</span> <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">server</span><span class="p">)</span>
    <span class="n">sthread</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="kc">True</span>                       <span class="c1"># don't wait for server thread</span>
    <span class="n">sthread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>                             <span class="c1"># do wait for children to exit</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> 
         <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">client</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="s1">'client</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">i</span><span class="p">,))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>client got: [server got: [client4]]
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="4-Signals">4 Signals<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#4-Signals">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [31]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># # catch signals in Python; pass signal number N as a command-line arg,</span>
<span class="c1"># # use a "kill -N pid" shell command to send this process a signal;  most</span>
<span class="c1"># # signal handlers restored by Python after caught (see network scripting</span>
<span class="c1"># # chapter for SIGCHLD details); on Windows, signal module is available,</span>
<span class="c1"># # but it defines only a few signal types there, and os.kill is missing;</span>

<span class="kn">import</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">signal</span><span class="o">,</span> <span class="nn">time</span>
<span class="k">def</span> <span class="nf">now</span><span class="p">():</span> <span class="k">return</span> <span class="n">time</span><span class="o">.</span><span class="n">ctime</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">())</span>        <span class="c1"># current time string</span>

<span class="k">def</span> <span class="nf">onSignal</span><span class="p">(</span><span class="n">signum</span><span class="p">,</span> <span class="n">stackframe</span><span class="p">):</span>                <span class="c1"># python signal handler</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'Got signal'</span><span class="p">,</span> <span class="n">signum</span><span class="p">,</span> <span class="s1">'at'</span><span class="p">,</span> <span class="n">now</span><span class="p">())</span>     <span class="c1"># most handlers stay in effect</span>

<span class="n">signum</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="n">signum</span><span class="p">,</span> <span class="n">onSignal</span><span class="p">)</span>                  <span class="c1"># install signal handler</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span> <span class="n">signal</span><span class="o">.</span><span class="n">pause</span><span class="p">()</span>                       <span class="c1"># wait for signals (or: pass)</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [32]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># set and catch alarm timeout signals in Python; time.sleep doesn't play</span>
<span class="c1"># well with alarm (or signal in general in my Linux PC), so we call</span>
<span class="c1"># signal.pause here to do nothing until a signal is received;</span>

<span class="kn">import</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">signal</span><span class="o">,</span> <span class="nn">time</span>
<span class="k">def</span> <span class="nf">now</span><span class="p">():</span> <span class="k">return</span> <span class="n">time</span><span class="o">.</span><span class="n">asctime</span><span class="p">()</span>

<span class="k">def</span> <span class="nf">onSignal</span><span class="p">(</span><span class="n">signum</span><span class="p">,</span> <span class="n">stackframe</span><span class="p">):</span>                 <span class="c1"># python signal handler</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'Got alarm'</span><span class="p">,</span> <span class="n">signum</span><span class="p">,</span> <span class="s1">'at'</span><span class="p">,</span> <span class="n">now</span><span class="p">())</span>       <span class="c1"># most handlers stay in effect</span>

<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'Setting at'</span><span class="p">,</span> <span class="n">now</span><span class="p">())</span>
    <span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGALRM</span><span class="p">,</span> <span class="n">onSignal</span><span class="p">)</span>       <span class="c1"># install signal handler</span>
    <span class="n">signal</span><span class="o">.</span><span class="n">alarm</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>                               <span class="c1"># do signal in 5 seconds</span>
    <span class="n">signal</span><span class="o">.</span><span class="n">pause</span><span class="p">()</span>                                <span class="c1"># wait for signals</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>('Setting at', 'Tue May 20 15:58:01 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:06 2014')
('Setting at', 'Tue May 20 15:58:06 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:11 2014')
('Setting at', 'Tue May 20 15:58:11 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:16 2014')
('Setting at', 'Tue May 20 15:58:16 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:21 2014')
('Setting at', 'Tue May 20 15:58:21 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:26 2014')
('Setting at', 'Tue May 20 15:58:26 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:31 2014')
('Setting at', 'Tue May 20 15:58:31 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:36 2014')
('Setting at', 'Tue May 20 15:58:36 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:41 2014')
('Setting at', 'Tue May 20 15:58:41 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:46 2014')
('Setting at', 'Tue May 20 15:58:46 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:51 2014')
('Setting at', 'Tue May 20 15:58:51 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:58:56 2014')
('Setting at', 'Tue May 20 15:58:56 2014')
('Got alarm', 14, 'at', 'Tue May 20 15:59:01 2014')
('Setting at', 'Tue May 20 15:59:01 2014')
</pre>
</div>
</div>

<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansi-red-intense-fg ansi-bold">---------------------------------------------------------------------------</span>
<span class="ansi-red-intense-fg ansi-bold">KeyboardInterrupt</span>                         Traceback (most recent call last)
<span class="ansi-green-intense-fg ansi-bold">&lt;ipython-input-32-120c29997daa&gt;</span> in <span class="ansi-cyan-fg">&lt;module&gt;</span><span class="ansi-blue-intense-fg ansi-bold">()</span>
<span class="ansi-green-fg">     13</span>     signal<span class="ansi-yellow-intense-fg ansi-bold">.</span>signal<span class="ansi-yellow-intense-fg ansi-bold">(</span>signal<span class="ansi-yellow-intense-fg ansi-bold">.</span>SIGALRM<span class="ansi-yellow-intense-fg ansi-bold">,</span> onSignal<span class="ansi-yellow-intense-fg ansi-bold">)</span>       <span class="ansi-red-intense-fg ansi-bold"># install signal handler</span>
<span class="ansi-green-fg">     14</span>     signal<span class="ansi-yellow-intense-fg ansi-bold">.</span>alarm<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-cyan-intense-fg ansi-bold">5</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>                               <span class="ansi-red-intense-fg ansi-bold"># do signal in 5 seconds</span>
<span class="ansi-green-intense-fg ansi-bold">---&gt; 15</span><span class="ansi-yellow-intense-fg ansi-bold">     </span>signal<span class="ansi-yellow-intense-fg ansi-bold">.</span>pause<span class="ansi-yellow-intense-fg ansi-bold">(</span><span class="ansi-yellow-intense-fg ansi-bold">)</span>                                <span class="ansi-red-intense-fg ansi-bold"># wait for signals</span>

<span class="ansi-red-intense-fg ansi-bold">KeyboardInterrupt</span>: </pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="The-multiprocessing-Module">The multiprocessing Module<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#The-multiprocessing-Module">¶</a>
</h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="1-The-Basics:-Processes-and-Locks">1 The Basics: Processes and Locks<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#1-The-Basics:-Processes-and-Locks">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [33]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="sd">"""</span>
<span class="sd">multiprocess basics: Process works like threading.Thread, but </span>
<span class="sd">runs function call in parallel in a process instead of a thread;</span>
<span class="sd">locks can be used to synchronize, e.g. prints on some platforms;</span>
<span class="sd">starts new interpreter on windows, forks a new process on unix;</span>
<span class="sd">"""</span>

<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Lock</span>

<span class="k">def</span> <span class="nf">whoami</span><span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="n">lock</span><span class="p">):</span>
    <span class="n">msg</span> <span class="o">=</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">: name:</span><span class="si">%s</span><span class="s1">, pid:</span><span class="si">%s</span><span class="s1">'</span>
    <span class="k">with</span> <span class="n">lock</span><span class="p">:</span>
        <span class="nb">print</span><span class="p">(</span><span class="n">msg</span> <span class="o">%</span> <span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="vm">__name__</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">()))</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="n">lock</span> <span class="o">=</span> <span class="n">Lock</span><span class="p">()</span>
    <span class="n">whoami</span><span class="p">(</span><span class="s1">'function call'</span><span class="p">,</span> <span class="n">lock</span><span class="p">)</span>

    <span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">whoami</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="s1">'spawned child'</span><span class="p">,</span> <span class="n">lock</span><span class="p">))</span>
    <span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
    <span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>

    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
        <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">whoami</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">((</span><span class="s1">'run process </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">i</span><span class="p">),</span> <span class="n">lock</span><span class="p">))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>

    <span class="k">with</span> <span class="n">lock</span><span class="p">:</span>
        <span class="nb">print</span><span class="p">(</span><span class="s1">'Main process exit.'</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>('Got alarm', 14, 'at', 'Tue May 20 15:59:06 2014')
function call: name:__main__, pid:3257
spawned child: name:__main__, pid:5196
Main process exit.
run process 0: name:__main__, pid:5204
run process 1: name:__main__, pid:5205
run process 2: name:__main__, pid:5206
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="2-IPC-Tools:-Pipes,-Shared-Memory,-and-Queues">2 IPC Tools: Pipes, Shared Memory, and Queues<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2-IPC-Tools:-Pipes,-Shared-Memory,-and-Queues">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="2.1-multiprocessing-pipes">2.1 multiprocessing pipes<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.1-multiprocessing-pipes">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [ ]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1"># Use multiprocess anonymous pipes to communicate. Returns 2 connection</span>
<span class="c1"># object representing ends of the pipe: objects are sent on one end and</span>
<span class="c1"># received on the other, though pipes are bidirectional by default</span>

<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Pipe</span>

<span class="k">def</span> <span class="nf">sender</span><span class="p">(</span><span class="n">pipe</span><span class="p">):</span>
    <span class="n">pipe</span><span class="o">.</span><span class="n">send</span><span class="p">([</span><span class="s1">'spam'</span><span class="p">]</span> <span class="o">+</span> <span class="p">[</span><span class="mi">42</span><span class="p">,</span> <span class="s1">'eggs'</span><span class="p">])</span>
    <span class="n">pipe</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>

<span class="k">def</span> <span class="nf">talker</span><span class="p">(</span><span class="n">pipe</span><span class="p">):</span>
    <span class="n">pipe</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'bob'</span><span class="p">,</span> <span class="n">spam</span><span class="o">=</span><span class="mi">42</span><span class="p">))</span>
    <span class="n">reply</span> <span class="o">=</span> <span class="n">pipe</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
    <span class="nb">print</span> <span class="s1">'talk got:'</span><span class="p">,</span> <span class="n">reply</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="n">parentEnd</span><span class="p">,</span> <span class="n">childEnd</span> <span class="o">=</span> <span class="n">Pipe</span><span class="p">()</span>
    <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">sender</span><span class="p">,</span><span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">childEnd</span><span class="p">,))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
    <span class="nb">print</span> <span class="s1">'parent got:'</span><span class="p">,</span> <span class="n">parentEnd</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
    <span class="n">parentEnd</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>

    <span class="n">parentEnd</span><span class="p">,</span> <span class="n">childEnd</span> <span class="o">=</span> <span class="n">Pipe</span><span class="p">()</span>
    <span class="n">child</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">talker</span><span class="p">,</span><span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">parentEnd</span><span class="p">,))</span>
    <span class="n">child</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
    <span class="nb">print</span> <span class="s1">'parent got:'</span><span class="p">,</span> <span class="n">parentEnd</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
    <span class="n">parentEnd</span><span class="o">.</span><span class="n">send</span><span class="p">([</span><span class="n">x</span> <span class="o">*</span><span class="mi">2</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="s1">'spam'</span><span class="p">])</span>
    <span class="n">child</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
    <span class="nb">print</span> <span class="s1">'parent exit'</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="2.2-Shared-memory-and-globals">2.2 Shared memory and globals<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.2-Shared-memory-and-globals">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [37]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="sd">"""</span>
<span class="sd">Use multiprocess shared memory objects to communicate.</span>
<span class="sd">Passed objects are shared, but globals are not on Windows.</span>
<span class="sd">Last test here reflects common use case: distributing work.</span>
<span class="sd">"""</span>

<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Value</span><span class="p">,</span> <span class="n">Array</span>

<span class="n">procs</span> <span class="o">=</span> <span class="mi">3</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">0</span>    <span class="c1"># per-process globals, not shared </span>

<span class="k">def</span> <span class="nf">showdata</span><span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="n">val</span><span class="p">,</span> <span class="n">arr</span><span class="p">):</span>
    <span class="sd">"""</span>
<span class="sd">    print data values in this process</span>
<span class="sd">    """</span>
    <span class="n">msg</span> <span class="o">=</span> <span class="s1">'</span><span class="si">%-12s</span><span class="s1">: pid:</span><span class="si">%4s</span><span class="s1">, global:</span><span class="si">%s</span><span class="s1">, value:</span><span class="si">%s</span><span class="s1">, array:</span><span class="si">%s</span><span class="s1">'</span>
    <span class="nb">print</span><span class="p">(</span><span class="n">msg</span> <span class="o">%</span> <span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">count</span><span class="p">,</span> <span class="n">val</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="nb">list</span><span class="p">(</span><span class="n">arr</span><span class="p">)))</span>

<span class="k">def</span> <span class="nf">updater</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="n">arr</span><span class="p">):</span>
    <span class="sd">"""</span>
<span class="sd">    communicate via shared memory</span>
<span class="sd">    """</span>
    <span class="k">global</span> <span class="n">count</span>
    <span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span>                         <span class="c1"># global count not shared</span>
    <span class="n">val</span><span class="o">.</span><span class="n">value</span> <span class="o">+=</span> <span class="mi">1</span>                     <span class="c1"># passed in objects are</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span> <span class="n">arr</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [38]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="n">scalar</span> <span class="o">=</span> <span class="n">Value</span><span class="p">(</span><span class="s1">'i'</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>             <span class="c1"># shared memory: process/thread safe</span>
    <span class="n">vector</span> <span class="o">=</span> <span class="n">Array</span><span class="p">(</span><span class="s1">'d'</span><span class="p">,</span> <span class="n">procs</span><span class="p">)</span>         <span class="c1"># type codes from ctypes: int, double</span>

    <span class="c1"># show start value in parent process</span>
    <span class="n">showdata</span><span class="p">(</span><span class="s1">'parent start'</span><span class="p">,</span> <span class="n">scalar</span><span class="p">,</span> <span class="n">vector</span><span class="p">)</span>

    <span class="c1"># spawn child, pass in shared memory</span>
    <span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">showdata</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="s1">'child '</span><span class="p">,</span> <span class="n">scalar</span><span class="p">,</span> <span class="n">vector</span><span class="p">))</span>
    <span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">();</span> <span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>

    <span class="c1"># pass in shared memory updated in parent, wait for each to finish</span>
    <span class="c1"># each child sees updates in parent so far for args (but not global)</span>

    <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">loop1 (updates in parent, serial children)...'</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">procs</span><span class="p">):</span>
        <span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="n">scalar</span><span class="o">.</span><span class="n">value</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="n">vector</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">showdata</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">((</span><span class="s1">'process </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">i</span><span class="p">),</span> <span class="n">scalar</span><span class="p">,</span> <span class="n">vector</span><span class="p">))</span>
        <span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">();</span> <span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>

    <span class="c1"># same as prior, but allow children to run in parallel</span>
    <span class="c1"># all see the last iteration's result because all share objects</span>

    <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">loop2 (updates in parent, parallel children)...'</span><span class="p">)</span>
    <span class="n">ps</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">procs</span><span class="p">):</span>
        <span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="n">scalar</span><span class="o">.</span><span class="n">value</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="n">vector</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">showdata</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">((</span><span class="s1">'process </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">i</span><span class="p">),</span> <span class="n">scalar</span><span class="p">,</span> <span class="n">vector</span><span class="p">))</span>
        <span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
        <span class="n">ps</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">ps</span><span class="p">:</span> <span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>

    <span class="c1"># shared memory updated in spawned children, wait for each </span>

    <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">loop3 (updates in serial children)...'</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">procs</span><span class="p">):</span>
        <span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">updater</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">scalar</span><span class="p">,</span> <span class="n">vector</span><span class="p">))</span>
        <span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
        <span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
    <span class="n">showdata</span><span class="p">(</span><span class="s1">'parent temp'</span><span class="p">,</span> <span class="n">scalar</span><span class="p">,</span> <span class="n">vector</span><span class="p">)</span>
    
    <span class="c1"># same, but allow children to update in parallel</span>

    <span class="n">ps</span> <span class="o">=</span> <span class="p">[]</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">loop4 (updates in parallel children)...'</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">procs</span><span class="p">):</span>
        <span class="n">p</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">updater</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">scalar</span><span class="p">,</span> <span class="n">vector</span><span class="p">))</span>
        <span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
        <span class="n">ps</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">ps</span><span class="p">:</span> <span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
    <span class="c1"># global count=6 in parent only  </span>
    <span class="c1"># show final results here</span>
    <span class="c1"># scalar=12:  +6 parent, +6 in 6 children</span>
    <span class="n">showdata</span><span class="p">(</span><span class="s1">'parent end'</span><span class="p">,</span> <span class="n">scalar</span><span class="p">,</span> <span class="n">vector</span><span class="p">)</span> <span class="c1"># array[i]=8: +2 parent, +6 in 6 children</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>parent start: pid:3257, global:0, value:0, array:[0.0, 0.0, 0.0]

loop1 (updates in parent, serial children)...
child       : pid:5361, global:0, value:0, array:[0.0, 0.0, 0.0]
process 0   : pid:5369, global:1, value:1, array:[1.0, 0.0, 0.0]
process 1   : pid:5377, global:2, value:2, array:[1.0, 1.0, 0.0]

loop2 (updates in parent, parallel children)...
process 2   : pid:5385, global:3, value:3, array:[1.0, 1.0, 1.0]

loop3 (updates in serial children)...
process 0   : pid:5393, global:4, value:6, array:[2.0, 2.0, 2.0]
process 2   : pid:5397, global:6, value:6, array:[2.0, 2.0, 2.0]
process 1   : pid:5394, global:5, value:6, array:[2.0, 2.0, 2.0]
parent temp : pid:3257, global:6, value:9, array:[5.0, 5.0, 5.0]

loop4 (updates in parallel children)...
parent end  : pid:3257, global:6, value:12, array:[8.0, 8.0, 8.0]
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="2.3-Queues-and-subclassing">2.3 Queues and subclassing<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#2.3-Queues-and-subclassing">¶</a>
</h4>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [46]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span><span class="o">,</span> <span class="nn">Queue</span>
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">Queue</span> <span class="k">as</span> <span class="n">queue</span>
<span class="k">class</span> <span class="nc">Counter</span><span class="p">(</span><span class="n">Process</span><span class="p">):</span>
    <span class="sd">"""docstring for Counter"""</span>
    <span class="n">label</span> <span class="o">=</span> <span class="s1">' @'</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">Queue</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">start</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">post</span>  <span class="o">=</span> <span class="n">Queue</span>
        <span class="n">Process</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
    <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span>
            <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">state</span> <span class="o">+=</span> <span class="mi">1</span>
            <span class="nb">print</span> <span class="bp">self</span><span class="o">.</span><span class="n">label</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">pid</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">state</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">post</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">pid</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">state</span><span class="p">)</span>
        <span class="nb">print</span> <span class="bp">self</span><span class="o">.</span><span class="n">label</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">pid</span><span class="p">,</span> <span class="s1">'-'</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="nb">print</span> <span class="s1">'start'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">()</span>
    <span class="n">expected</span> <span class="o">=</span> <span class="mi">9</span>
    <span class="n">post</span> <span class="o">=</span> <span class="n">queue</span><span class="p">()</span>
    <span class="n">p</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="n">post</span><span class="p">)</span>
    <span class="n">q</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="n">post</span><span class="p">)</span>
    <span class="n">r</span> <span class="o">=</span> <span class="n">Counter</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span><span class="n">post</span><span class="p">)</span>
    <span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">();</span> <span class="n">q</span><span class="o">.</span><span class="n">start</span><span class="p">();</span> <span class="n">r</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>

    <span class="k">while</span> <span class="n">expected</span><span class="p">:</span>                             <span class="c1"># parent consumes data on queue</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)</span>                         <span class="c1"># this is essentially like a GUI,</span>
        <span class="k">try</span><span class="p">:</span>                                    <span class="c1"># though GUIs often use threads</span>
            <span class="n">data</span> <span class="o">=</span> <span class="n">post</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">block</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
        <span class="k">except</span> <span class="n">Queue</span><span class="o">.</span><span class="n">Empty</span><span class="p">:</span>
            <span class="nb">print</span> <span class="s1">'no data...'</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="nb">print</span> <span class="s1">'posted:'</span><span class="p">,</span> <span class="n">data</span>
            <span class="n">expected</span> <span class="o">-=</span> <span class="mi">1</span>

    <span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">();</span> <span class="n">q</span><span class="o">.</span><span class="n">join</span><span class="p">();</span> <span class="n">r</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>                <span class="c1"># must get before join putter</span>
    <span class="nb">print</span> <span class="s1">'finish'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">getpid</span><span class="p">(),</span> <span class="n">r</span><span class="o">.</span><span class="n">exitcode</span>   <span class="c1"># exitcode is child exit status</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>start 3257
no data...
no data...
no data...
no data...
no data...
no data... @ 5847 1

posted: @ 5848 101
 @ 5849 1001
 5848
posted: 5847
posted: 5849
no data...
no data...
posted: @ @ 5847 2
 5848 102
 5847
posted: @ 5849 1002
 5848
posted: 5849
no data...
no data...
no data...
posted: @ @ @ 5847 3
 5848 103
 5849 1003
 @ 5847 -
 @ 5848 -
 @ 5849 -
 5847
posted: 5848
posted: 5849
finish 3257 0
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="3-Starting-Independent-Programs">3 Starting Independent Programs<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#3-Starting-Independent-Programs">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [57]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="s2">"Use multiprocessing to start independent programs, os.fork or not"</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Process</span>

<span class="k">def</span> <span class="nf">runp</span><span class="p">(</span><span class="n">arg</span><span class="p">):</span>
    <span class="n">os</span><span class="o">.</span><span class="n">execlp</span><span class="p">(</span><span class="s1">'python'</span><span class="p">,</span> <span class="s1">'python'</span><span class="p">,</span> <span class="s1">'child.py'</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">arg</span><span class="p">))</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
        <span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">runp</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">i</span><span class="p">,))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
    <span class="nb">print</span><span class="p">(</span><span class="s1">'parent exit'</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>parent exit
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="4-process-pools">4 process pools<a class="anchor-link" href="python-parallel-system-tools-pp4e.html#4-process-pools">¶</a>
</h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [61]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="k">import</span> <span class="n">Pool</span>

<span class="k">def</span> <span class="nf">powers</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
    <span class="k">return</span> <span class="mi">2</span> <span class="o">**</span> <span class="n">x</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
    <span class="n">workers</span> <span class="o">=</span> <span class="n">Pool</span><span class="p">(</span><span class="n">processes</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>

    <span class="n">results</span> <span class="o">=</span> <span class="n">workers</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">powers</span><span class="p">,[</span><span class="mi">2</span><span class="p">]</span><span class="o">*</span><span class="mi">100</span><span class="p">)</span>
    <span class="nb">print</span> <span class="n">results</span><span class="p">[:</span><span class="mi">16</span><span class="p">]</span>
    <span class="nb">print</span> <span class="n">results</span><span class="p">[</span><span class="o">-</span><span class="mi">2</span><span class="p">:]</span>

    <span class="n">results</span> <span class="o">=</span> <span class="n">workers</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">powers</span><span class="p">,</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
    <span class="nb">print</span> <span class="n">results</span><span class="p">[:</span><span class="mi">16</span><span class="p">]</span>
    <span class="nb">print</span> <span class="n">results</span><span class="p">[</span><span class="o">-</span><span class="mi">2</span><span class="p">:]</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>[4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4]
[4, 4]
[1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
[256, 512]
</pre>
</div>
</div>

</div>
</div>

</div>
    </div>
  </div>

    </div>
    <aside class="postpromonav"><nav><ul itemprop="keywords" class="tags">
<li><a class="tag p-category" href="../categories/parallel-programming-with-python.html" rel="tag">Parallel Programming with Python</a></li>
            <li><a class="tag p-category" href="../categories/python.html" rel="tag">Python</a></li>
        </ul>
<ul class="pager hidden-print">
<li class="previous">
                <a href="kivy-ch2-paint-app.html" rel="prev" title="kivy-ch2-paint-app">Previous post</a>
            </li>
            <li class="next">
                <a href="kivy-ch3-sound-recorder-for-android.html" rel="next" title="kivy-ch3-sound-recorder-for-android">Next post</a>
            </li>
        </ul></nav></aside><script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"> </script><script type="text/x-mathjax-config">
MathJax.Hub.Config({
    tex2jax: {
        inlineMath: [ ['$','$'], ["\\(","\\)"] ],
        displayMath: [ ['$$','$$'], ["\\[","\\]"] ],
        processEscapes: true
    },
    displayAlign: 'center', // Change this to 'center' to center equations.
    "HTML-CSS": {
        styles: {'.MathJax_Display': {"margin": 0}}
    }
});
</script></article>
</div>
        <!--End of body content-->

        <footer id="footer">
            Contents © 2017         <a href="mailto:muxuezi@gmail.com">Tao Junjie</a> - Powered by         <a href="https://getnikola.com" rel="nofollow">Nikola</a>         
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0">
<img alt="Creative Commons License BY-NC-SA" style="border-width:0; margin-bottom:12px;" src="http://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png"></a>
            
        </footer>
</div>
</div>


            <script src="../assets/js/all-nocdn.js"></script><script>$('a.image-reference:not(.islink) img:not(.islink)').parent().colorbox({rel:"gal",maxWidth:"100%",maxHeight:"100%",scalePhotos:true});</script><!-- fancy dates --><script>
    moment.locale("en");
    fancydates(0, "YYYY-MM-DD HH:mm");
    </script><!-- end fancy dates --><script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-51330059-1', 'auto');
  ga('send', 'pageview');

</script>
</body>
</html>
